Skip to content

[pull] master from DataDog:master#567

Merged
pull[bot] merged 5 commits into
ConnectionMaster:masterfrom
DataDog:master
May 28, 2026
Merged

[pull] master from DataDog:master#567
pull[bot] merged 5 commits into
ConnectionMaster:masterfrom
DataDog:master

Conversation

@pull

@pull pull Bot commented May 28, 2026

Copy link
Copy Markdown

See Commits and Changes for more details.


Created by pull[bot] (v2.0.0-alpha.4)

Can you help keep this open source service alive? 💖 Please sponsor : )

AAraKKe and others added 5 commits May 28, 2026 08:43
…23850)

* Wait for the gamesim_primary index to be online before running tests

test_index_stats_metrics asserts metrics tagged with index_name:gamesim_primary,
which the Couchbase indexer only reports once the bundled GSI index for
gamesim-sample finishes building. Add a WaitFor against /api/v1/stats on port
9102 (Couchbase 7+) so the environment fixture blocks until the indexer is
actually reporting that keyspace.

Also fix the inverted exit condition in load_sample_bucket(): the loop was
breaking the moment the task appeared in /pools/default/tasks instead of
when it disappeared, so it returned while the sample load was still running.

* Confirm gamesim_primary build completes before yielding environment

Three follow-ups from review:

- Check `initial_build_progress == 100` on the keyspace; the indexer also
  publishes stats while an index is still building.
- Use `raise_for_status()` so 401/404/5xx surface immediately instead of
  collapsing into a 60s WaitFor timeout.
- Drop the >=7 conditional around the new WaitFor; the test matrix only
  covers 7.x and the test that needs the index is itself gated on 7+.
- Bail out of load_sample_bucket if the install response carries no
  matching task entry, instead of spinning on `None == None`.

* Require every gamesim_primary keyspace to report build progress 100

* Condense gamesim_primary_index_ready docstring to a one-liner

* Clarify gamesim_primary_index_ready and log keyspaces on the negative path

- Refactor to a list comprehension that separates 'no keyspace yet' from
  'still building' so each retry path is self-documenting.
- Print the keyspaces seen on the negative path so a future WaitFor timeout
  doesn't leave a debugging dead end.
- Document why load_sample_bucket bails out when the install response carries
  no matching task — the next gamesim_primary_index_ready WaitFor is the real
  readiness gate.
* Honor source ref in release dispatch

Keep release workflow tooling checked out separately from the source tree being released. This lets the workflow use current setup actions and release scripts while ddev tags and validates the requested source-repo-ref.

Add source-repo-branch to manual release dispatches so stable versus pre-release validation follows the branch that contains source-repo-ref instead of the branch used to launch the workflow.

* align source-repo-branch description with release-trigger.yml
* [LOGSP-29] Add bypass annotations for linter checks

A linter rule is being added or changed and some integrations are expected to remain non-compliant.
Add permanent bypass annotations so CI stays green and does not fail for new PRs.

* fix(logs-linter): bypass date-remapper-parse-failure-checks for known false positives

These integrations either use log samples that intentionally lack
timestamp source fields, or rely on date formats the test harness
cannot parse. The date-remapper is functioning correctly in production
for all affected integrations.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* Trigger CI

---------

Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* WIP

* Change orchestrator message

* Add changelog and fix tests

* Adjust Datadog code coverage gates

* Sync Datadog code coverage services

* Add integrations that don't pass threshold

* Validate stale Datadog coverage services

Detect services present in the Datadog Code Coverage config but no longer expected by ddev validate ci, and remove them when running with --sync.

* Lower cilium threshold

* Validate code coverage gates

* Validate duplicate coverage services

* Remove tests from coverage config

* Exclude tests from coverage services

* Lower coverage gates for migration exceptions
* Store ClickHouse advanced-queries metric definitions as JSON

The four advanced_queries Python modules shipped ~250 KB of redundant
dict literals (per-entry 'name' was always '<prefix>.<key>', and every
entry repeated its type). Move that data to compact per-system-table
JSON files under datadog_checks/clickhouse/data/ and build the
QueryManager-shaped dicts at runtime.

The check registers a check_initializations callable so the JSON files
are parsed once on the first check run. Module attributes like
advanced_queries.SystemMetrics remain available through __getattr__
backed by the same cache, so tests that read those attributes directly
keep working.

The metric generator emits the new JSON format directly; the three
system_*.tpl templates and the four old Python modules are removed.

* Add changelog entry

* Simplify advanced_queries loader and tighten the generator

- Replace the initializer(check) factory with a plain warm_cache callable;
  the closure never depended on the check instance.
- Restore __all__ on the package so the public surface is explicit.
- Rename _cache to cache per the AGENTS.md module-name rule.
- Mirror Python's default AttributeError format in __getattr__.
- Tighten return-type annotations on load, _build_items, and __getattr__.
- Raise loudly in generate_queries when a metric type appears with mixed
  scaled/unscaled entries instead of silently producing a wrong container.
- Reclass the changelog from .changed to .fixed: the refactor preserves
  runtime behaviour byte-for-byte, so a patch bump is the right semver.

* Tighten the advanced_queries loader

- Guard warm_cache with an explicit "key not in cache" check so the
  JSON files are read at most once per process, even when __getattr__
  populated the cache first.
- Wrap load() exceptions as RuntimeError so a missing or malformed
  data file produces an actionable message in the check-init failure.
- Discriminate the two JSON shapes on positive presence of "columns"
  rather than absence of "items".
- Reclass the changelog as .changed: this is a significant internal
  refactor, not a bug fix, so .fixed was misleading.

* Drop system_errors from the generator and widen load() error wrap

- Wrap KeyError in load()'s error path so a malformed JSON file (one
  that parses but is missing expected keys) raises the same context-rich
  RuntimeError as a missing or invalid file.
- Remove SYSTEM_ERRORS_SPEC and generate_system_errors() from the
  generator. The system_errors data is static; the committed
  data/system_errors.json is the single source of truth, and the other
  three queries continue to be generator-driven from ClickHouse source.

* Tighten the advanced_queries loader and rename the generator's Template

- Comment the load() branch that handles system_errors so the asymmetry
  is named at the call site instead of requiring readers to audit the
  JSON files.
- Widen load()'s except tuple with TypeError and AttributeError so a
  malformed JSON shape (e.g. items shipped as a list) still raises the
  wrapped RuntimeError with the file name rather than leaking the bare
  underlying exception.
- Narrow _build_items's compact parameter type to
  dict[str, list[str] | dict[str, str]] to mirror the producer
  annotation in generate_metrics.py.
- Rename the generator's Template dataclass to FileTemplate so it
  doesn't silently shadow string.Template if the stdlib import is ever
  reintroduced.

* Rename module cache to _cache and drop the one-member Templates enum

- Rename advanced_queries.cache to _cache so the underscore signals
  it as module-internal mutable state (PEP 8 module-private). The
  module's __all__ already advertises only the four System* names.
- Rephrase the load() inline comment around the columns shortcut so it
  names the discriminating shape rather than the system_errors file,
  which would mislead if a second verbatim file were added later.
- Replace the single-member Templates enum in the generator with a
  TESTS_METRICS_TEMPLATE constant; the enum was a vestige from when it
  held the three QUERY_* templates that now live in QUERY_SPECS.

* Test the advanced_queries loader directly

Adds tests/test_advanced_queries.py covering the new loader logic:
- module-level __getattr__ resolution + caching
- compact format: source/match column shape, sorted items, name
  derivation including the dotted-key edge case (jemalloc.epoch)
- verbatim format: system_errors columns pass through with the
  boolean: true tag preserved
- RuntimeError wrap on every malformed-JSON path the load() except
  tuple is meant to cover (missing file, invalid JSON, items as list,
  items as scalar, missing required keys) with the cause chain
  preserved
- warm_cache populates every known name and is idempotent

* Move changelog to fixed

* Scope the advanced_queries loader to bulk match queries

The advanced_queries package is now organised around one named pattern:
the SQL-returns-(value, metric_name)-and-dispatches-via-lookup-table
shape that SystemEvents, SystemMetrics, and SystemAsynchronousMetrics
all share. The compact JSON files exist specifically to compress that
pattern.

- Rename load() to load_match_query(); _build_items() to
  _expand_match_items(); NAMES to MATCH_QUERIES; _cache to
  _match_query_cache. Names now say what the loader does.
- Inline SystemErrors as a plain Python literal in __init__.py. Its
  shape doesn't fit the bulk-match pattern, so the JSON compression has
  nothing to compress; data/system_errors.json is removed and the
  verbatim-columns branch in load() goes away with it.
- Add a top-level docstring that describes the JSON schema, names the
  generator that produces the files, and points operators at the
  hatch run metrics:generate command.
- Add clickhouse/AGENTS.md (with a CLAUDE.md @AGENTS.md indirection)
  giving anyone opening this directory a short orientation note plus
  the "don't hand-edit the JSON files" warning that JSON has no comment
  syntax to carry.
- Update tests/test_advanced_queries.py for the rename and add coverage
  that SystemErrors stays out of the match-query cache.

Runtime dict shape and metric names are byte-identical to before;
verified by diffing the four module-attribute dumps against the
pre-refactor master.
@pull pull Bot locked and limited conversation to collaborators May 28, 2026
@pull pull Bot added the ⤵️ pull label May 28, 2026
@pull pull Bot merged commit d6365cc into ConnectionMaster:master May 28, 2026
1 check passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants